Skip to content

fix(engine): keep runner-owned checkouts across control-plane restarts - #1188

Open
wcy12378 wants to merge 1 commit into
Tencent:mainfrom
wcy12378:codex/runner-staging-owner-recovery
Open

wcy12378 wants to merge 1 commit into
Tencent:mainfrom
wcy12378:codex/runner-staging-owner-recovery

Conversation

@wcy12378

Copy link
Copy Markdown

fix(engine): keep runner-owned checkouts across control-plane restarts

This is the runner owner recovery slice of #1079. When the platform container is recreated its hostname changes, leaving runs bound to a host that no longer exists. For runs that execute on a runner, the checkout lives on the runner — the platform-side directory only stages prompts and artifacts — so a stale platform host must not block the run forever.

Problem

workspaceBlocked treated a dead recorded host as blocking whenever the run had a default execution runner. But a runner-owned checkout survives a control-plane restart (the runner keeps its mounts), so such a run was stuck: its approval gate could not be approved and its workspace was reported as lost even though no work was actually lost.

Change

Distinguish platform-owned from runner-owned checkouts:

  • internal/engine/run_recovery.go
    • workspaceBlocked no longer returns blocking for a dead recorded host when the run has an execution runner; the runner keeps its checkout.
    • workspaceLost only reports a checkout as lost for platform-owned runs (defaultExecutionRunner == ""), so cloud runs keep their existing dead-host protection.
  • internal/engine/run_environment.go
    • adoptForeignWorkspace lets a shared-workspace runner run continue on its runner instead of reporting the workspace as lost with the dead host.

Safety is preserved by the existing dispatch rules: once nodes have changed code, a run refuses to move to another machine (validateExecutionReconnect, bindPoolRunner), so continuing on the original runner cannot drop unpushed work. Live foreign hosts and same-host other-root binds still block takeover.

Verification (Linux/amd64, Docker python:3.12)

  • New RED→GREEN tests in internal/engine/workspace_runner_owner_test.go:
    • TestRunnerGateRecoversAfterPlatformHostChange — approving the gate after the platform host changed succeeds; the run continues on its runner and the unpushed checkout files are kept byte-for-byte.
    • TestRunnerWorkspaceRebindsWithoutReportingLoss — the next node rebinds the platform staging without reporting the workspace as lost.
    • TestRunnerGateStaysBlockedByLiveForeignHost / TestRunnerGateStaysBlockedBySameHostOtherRoot — live foreign / same-host other-root binds stay blocking.
    • TestRunnerReconnectKeepsTheOriginalRunner — a runner run never moves to another machine while the original is unreachable.
    • TestCloudRunStillTreatsDeadHostAsLost — platform-owned checkouts keep their dead-host protection.
    • All six PASS. The two behavior-changing tests fail on the unmodified baseline with the same wording, confirming RED→GREEN.
  • Full engine regression: the patched set matches the baseline stable set (environment-limited gate MCP binary failures); no new failure beyond baseline. go vet and public-content checks pass.

Scope / notes

Related

Closes part of #1079 (runner owner recovery); does not close the whole issue.

When the platform container is recreated its hostname changes, leaving runs
bound to a host that no longer exists. For runs that execute on a runner the
checkout lives on the runner, not the platform; the platform-side directory
only stages prompts and artifacts. The stale host must not block such a run
forever.

Distinguish platform-owned from runner-owned checkouts:
- workspaceBlocked no longer treats a dead recorded host as blocking when the
  run has an execution runner; the runner keeps its checkout.
- workspaceLost only reports a checkout as lost for platform-owned runs
  (no execution runner), so cloud runs keep their dead-host protection.
- adoptForeignWorkspace lets a shared-workspace runner run continue on its
  runner instead of reporting the workspace as lost.

Safety is preserved by the existing dispatch rules: once nodes have changed
code, a run refuses to move to another machine (validateExecutionReconnect,
bindPoolRunner), so continuing on the original runner cannot drop unpushed
work. Live foreign hosts and same-host other-root binds still block takeover.

Related to Tencent#1079.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant